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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address » 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )□ Responsive to communication(s) filed on . 

2a)D This action is FINAL. 2b)IS This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 
closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-42 is/are pending in the application. 
4a) Of the above claim(s) 35-42 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) EI Claim(s) 1-6,9-16.19-26 and 29-32 is/are rejected. 
7M Claim(s) 5-8.16-18.26-28 and 31-34 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) ^ The specification is objected to by the Examiner. 

10) 13 The drawing(s) filed on 27 March 2000 is/are: a)E3 accepted or b)D objected to by the Examiner. 
Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

1 1) D The proposed drawing correction filed on is: a)Q approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 

* See the attached detailed Office action for a list of the certified copies not received. 

14) ^ Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application), 
a) □ The translation of the foreign language provisional application has been received. 

15) Q Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 

Attachment(s) 

1) 03 Notice of References Cited (PTO-892) 4) Interview Summary (PTO-41 3) Paper No(s). . 

2) n Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) O Notice of Informal Patent Application (PTO-152) 

3) (3 Information Disclosure Statement(s) (PTO-1449) Paper No(s) 3 . 6) Q Other: 

t 

U.S. Patent and Trademark Office — — 

PTO-326 (Rev. 04-01 ) Office Action Summary Part of Paper No. 8 
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DETAILED ACTION 
Election/Restrictions 

1. Restriction to one of the following inventions is required under 35 U.S.C. 121: 

I. Claims 1-34, drawn to face recognition process, classified in class 382, subclass 
118. 

II. Claims 35-43, drawn to a neural network ensemble, classified in class 382, 
subclass 156. 

2. Inventions I and II are related as combination and subcombination. Inventions in this 
relationship are distinct if it can be shown that (1) the combination as claimed does not require 
the particulars of the subcombination as claimed for patentability, and (2) that the 
subcombination has utility by itself or in other combinations (MPEP § 806.05(c)). In the instant 
case, the combination as claimed does not require the particulars of the subcombination as 
claimed because the combination deals with face and pose recognition. The subcombination has 
separate utility such as automatic pattern recognition tasks. 

3. During a telephone conversation with Richard Lyon on 2/5/03 a provisional election was 
made without traverse to prosecute the invention of I, claims 1-34. Affirmation of this election 
must be made by applicant in replying to this Office action. Claims 35-42 are withdrawn from 
further consideration by the examiner, 37 CFR 1 .142(b), as being drawn to a non-elected 
invention. 
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Specification 



4. The abstract of the disclosure is objected to because "persons" should be changed to 
"person's" on line 1 1 . Correction is required. See MPEP § 608.01(b). 



5. Claims 5, 6 5 16, 26, and 31-33 are objected to because of the following informalities: 
"Clam 5" should be changed to "Claim 5" on page 50, line 19. Regarding claims 6, 16, and 26, 
"PC A" should be replaced with "principal components analysis (PC A)" on page 51, line 8; page 
56, line 1; and page 60, line 26. Regarding claims 31-33, "DVC" should be changed to "DCV" 
throughout the claims. Appropriate correction is required. 



6. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

7. Claims 9 and 32 are rejected under 35 U.S.C. 112, second paragraph, as being indefinite 
for failing to particularly point out and distinctly claim the subject matter which applicant 
regards as the invention. 

Claim 9 recites the limitation "the neural network ensemble" in line 3. There is 
insufficient antecedent basis for this limitation in the claim. 

Claim 32 recites the limitation "the fusing neural network" in line 7. There is insufficient 
antecedent basis for this limitation in the claim. 



Claim Objections 



Claim Rejections - 35 USC § 112 
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Claim Rejections - 35 USC § 103 



8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

9. Claims 1-6, 9-16, 19-26, and 29 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Turk et al. (5,164,992) in view of Niyogi et al. (6,144,755). 

Regarding claim 1, Turk et al. ("Turk") discloses a face recognition process including 
creating a reference set or "database" of a plurality of model image characterizations, each of 
which represents the face of a known person that it is desired to identify in the input image (Col. 
3, lines 37-48) and comparing a region depicting the face of a person, which has been extracted 
from the input image and characterized in a manner similar to the plurality of model images, to 
the plurality of model image characterizations (Col. 4, lines 48-68 and Col. 5, lines 1-3). Turk 
further discloses ascertaining which of the plurality of model image characterizations most 
closely matches the similarly characterized input image region and designating the input image 
region to be the person associated with the most closely matching model image characterization 
if a degree of similarity exceeds a prescribed threshold (Col. 5, lines 3-12). Turk does not 
recognize creating a database that includes a person's face pose. However, Niyogi et al. 
("Niyogi") teaches that it is known to create a training set or "database" associated with a 
person's face pose (Col. 3, lines 22-28). Therefore, it would have been obvious to one of 
ordinary skill in the art at the time of the invention to have modified the database disclosed by 
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Turk to include a person's face pose, as taught by Niyogi, in order to analyze an image of a 
person in different positions (Col. 1, lines 21-24). 

Regarding claim 2, Turk discloses specifying the person designated as corresponding to 
the input image region (Col. 1 , lines 61-64). Turk does not disclose specifying that the person 
has the face pose associated with the most closely matching model image characterization. 
However, Niyogi teaches that it is known to specify the pose associated with the most closely 
matching model (Abstract, lines 15-16). Therefore, it would have been obvious to one of 
ordinary skill in the art at the time of the invention to have modified the specification of the 
person as disclosed by Turk to include specifying the face pose most closely matching the model 
image characterization, as taught by Niyogi, in order to more completely characterize the input 
image. 

Regarding claim 3, Turk discloses designating the input image region to be an unknown 
person determined by a prescribed threshold based on the degree of similarity between the 
characterized input region and the most closely matching model image characterization does not 
exceed the prescribed threshold (Col. 5, lines 8-12). 

Regarding claim 9, the arguments analogous to those presented above for claim 3 are 
applicable to claim 9. Turk discloses the implementation of a neural network to identify an 
unknown person (Col. 10, lines 23-28) which would thereby entail training and employing the 
neural network. 

Regarding claim 4, Turk discloses using a neural network ensemble to identify the person 
associated with the characterized input image region (Col. 10, lines 23-28) which would thereby 



Application/Control Number: 09/536,820 Page 6 

Art Unit: 2623 

entail training and employing the neural network. The arguments analogous to those presented 
above for claim 2 are applicable to claim 4. 

Regarding claim 5, Turk discloses preparing each model image characterization from a 
model image depicting the face of a known person that it is desired to identify in the input image 
by extracting the portion of the model image depicting the face 8 (Figure 1) and normalizing the 
extracted portion by resizing it to a prescribed scale if not already at a prescribed scale (Col. 10, 
lines 10-12). Turk does not disclose adjusting or cropping the region. However, Niyogi teaches 
that it is known to normalize the extracted portion by adjusting the region so that the head 2 
which includes the eye locations of the depicted subject fall within a prescribed area 410 (Figure 
4). Niyogi also teaches that it is known to crop the extracted portion of the model image by 
eliminating unneeded portions of the image not specifically depicting part of the face of the 
subject to create a model face image (Col. 4, lines 21-30). Therefore, it would have been 
obvious to one of ordinary skill in the art at the time of the invention to have modified the 
preparing of the model disclosed by Turk to include adjusting and cropping, as taught by Niyogi, 
in order to eliminate erroneous results. 

Regarding claim 6, Niyogi discloses categorizing the model face images by assigning 
each to one of a set of pose ranges (Figure 2) into which its associated face pose falls (Col. 3, 
lines 32-34). Niyogi further discloses choosing a prescribed number of model face images of 
each person being modeled which have been assigned to the selected pose range (Col. 3, lines 
49-50). Niyogi discloses using PC A (Col. 4, lines 2-4), but does not explicitly state the 
described details. However, Turk teaches that it is known to concatenate each of the chosen 
model face images to create a respective dimensional column vector for each (Col. 3, lines 49- 
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65), compute a covariance matrix from the DCVs (Col. 4, lines 1-7), calculate eigenvectors and 
the corresponding eigenvalues from the covariance matrix (Col. 4, lines 3-7), rank the 
eigenvalues in descending order and identify a prescribed number of the top eigenvalues (Col. 4, 
lines 30-37), use the eigenvectors corresponding to the identified eigenvalues to for the rows of a 
basis vector matrix (Col. 6, lines 57-60), and multiplying each DCV by each BVM to produce a 
set of PC A coefficient vectors for each model face image 104 (Figure 2). Therefore, it would 
have been obvious to one of ordinary skill in the art at the time of the invention to have modified 
the use of PCA disclosed by Niyogi to include the details mentioned above, as taught by Turk, in 
order to define the variation among the face images (Col. 4, lines 3-7). 

Regarding claim 10, Turk discloses a face recognition system (Abstract) that captures 
model images, each of which depicts at least one person of known identity (Col. 3, lines 37-48), 
captures the input image (Col. 3, lines 17-25), locates and extracts regions within the input 
images, each of which depicts the face of a person (Col. 1, line 68 and Col. 2, lines 1-4), and for 
each face region extracted from the input image, compares the face region to the reference set to 
identify the person depicted in the input image face region (Col. 1, lines 61-64). Turk does not 
recognize using a computer or determining the face pose. However, Niyogi teaches that it is 
known to include a computer 10 and a computer program comprising program modules 
executable by the computing device (Col. 3, lines 9-20). Niyogi also teaches determining the 
face pose for each of the face regions extracted from the model images (Col. 3, lines 22-27) and 
categorizing each face region by assigning each to one of a set of pose ranges (Figure 2) into 
which its associated face pose falls (Col. 3, lines 32-34). Therefore, it would have been obvious 
to one of ordinary skill in the art at the time of the invention to have modified the face 
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recognition system disclosed by Turk to include determining the face pose, as taught by Niyogi, 
in order to analyze an image of a person in different positions (Col 1, lines 21-24). 

Regarding claim 1 1, the arguments analogous to those presented above for claim 1 are 
applicable to claim 1 1 . Turk discloses assessing the degree of similarity between the input image 
face region and the reference set (Col. 4, lines 65-68 and Col. 5, lines 1-12). 

Regarding claim 12, the arguments analogous to those presented above for claim 2 are 
applicable to claim 12. 

Regarding claim 13, the arguments analogous to those presented above for claim 3 are 
applicable to claim 13. 

Regarding claim 14, the arguments analogous to those presented above for claim 4 are 
applicable to claim 14. 

Regarding claim 15, the arguments analogous to those presented above for claim 5 are 
applicable to claim 15. 

Regarding claim 16, the arguments analogous to those presented above for claim 6 are 
applicable to claim 16. Turk does not disclose repeating the actions for each pose. However, it 
would have been obvious to one of ordinary skill in the art at the time of the invention to have 
modified the training disclosed by Turk to include repeating for each pose taught by Niyogi in 
order to provide training associated with each pose. 

Regarding claim 19, the arguments analogous to those presented above for claim 9 are 
applicable to claim 19. 

Regarding claim 20, the arguments analogous to those presented above for claim 10 are 
applicable to claim 20. The claim is drawn to a computer-readable storage medium. While 
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Niyogi does not appear to explicitly mention a computer-readable storage medium, this would 
have been obvious in light of Niyogfs disclosure. Note, for example, Niyogi's disclosure 
frequently mentions CPU (e.g., Col. 2, line 66), thereby establishing his system as being or 
relating to a computer-based system. 

Regarding claim 21, the arguments analogous to those presented above for claim 1 1 are 
applicable to claim 21. 

Regarding claim 22, the arguments analogous to those presented above for claim 12 are 
applicable to claim 22. 

Regarding claim 23, the arguments analogous to those presented above for claim 13 are 
applicable to claim 23. 

Regarding claim 24, the arguments analogous to those presented above for claim 14 are 
applicable to claim 24. 

Regarding claim 25, the arguments analogous to those presented above for claim 15 are 
applicable to claim 25. 

Regarding claim 26, the arguments analogous to those presented above for claim 16 are 
applicable to claim 26. 

Regarding claim 29, the arguments analogous to those presented above for claim 9 are 
applicable to claim 29. 

10. Claims 30 and 31 are rejected under 35 U.S.C. 103(a) as being unpatentable over Turk et 
al. (5,164,992) in view of Niyogi et al (6,144,755) as applied to claiml above, and further in 
view of Kung et al. (5,850,470). 
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Regarding claim 30, the arguments analogous to those presented above for claim 4 are 
applicable to claim 30. Turk and Niyogi do not appear to expressly state a network ensemble 
comprising a first stage having a plurality of classifiers and a neural network as its second stage. 
However, Kung et al. ("Kung") teaches that it is known to use a network ensemble with a first 
stage having a plurality of classifiers and a neural network as its second stage (Figure 2a). 
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the 
invention to have modified the network ensemble disclosed by Turk and Niyogi to include the 
two stages consisting of classifiers and a neural network, as taught by Kung, in order to perform 
the classification task. 

Regarding claim 31, the arguments analogous to those presented above for claims 5 and 6 
are applicable to claim 3 1 . 

Allowable Subject Matter 

11. Claims 7, 8, 17, 18,27,28, and 32-34 are objected to as being dependent upon a rejected 
base claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 

Conclusion 

12. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

U.S. Pat. No. 6,292,575 to Bortolussi et al. for real-time facial recognition and 
verification system; 
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U.S. Pat. No. 5,497,430 to Sadovnik et al. for image recognition using invariant feature 
signals; and 

U.S. Pat. No. 6,463,163 to Kresch for face detection using candidate image region 
selection. 



1 3 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Virginia M Kibler whose telephone number is (703) 306-4072. 
The examiner can normally be reached on Mon. - Thurs. 8:00 - 5:30 and every other Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on (703) 308-6604. The fax phone numbers for the 
organization where this application or proceeding is assigned are (703) 872-9314 for regular 
communications and (703) 872-93 14 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703) 306-0377. 
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